Table data structures, and particularly tables in database management systems, are used to store large amounts of data. The demand for efficient data storage for a variety of data intensive applications continues to grow. However, for many such data intensive applications, table data structures have been assumed to be an inappropriate mechanism for storing much of the data generated or obtained by those applications. Furthermore, there appears to be little appreciation that the paradigms associated with table data structures would be very useful in those applications.
For instance, the web pages downloaded during a crawl of the World Wide Web (WWW) are typically stored as a set of files, not as entries in a table or a database table. Similarly, RSS feeds (which can be considered to be documents) downloaded by RSS feed aggregators and other computers are typically stored as files. Storing web content, RSS feed content and the like as individual files is traditional and the mechanisms for managing and accessing such files are well established. The present invention, on the other hand, provides a data model and a set of data management processes and mechanisms for storing large amounts of data in one or more table data structures, thereby providing an alternative to the traditional model of storing individual content items in individual files.